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ABSTRACT 


The issue of how to support situation awareness among crews of modem commercial 
aircraft is becoming especially important with the introduction of automation in the form of 
sophisticated flight management computers and expert systems designed to assist the crew. 
In this paper, cognitive theories are discussed that have relevance for the definition and 
measurement of situation awareness. These theories suggest that comprehension of the 
flow of events is an active process that is limited by the modularity of attention and memory 
constraints, but can be enhanced by expert knowledge and strategies. Three implications of 
this perspective for assessing and improving situation awareness are considered: (1) 
scenario variations are proposed that tax awareness by placing demands on attention; (2) 
experimental tasks and probes are described for assessing the cognitive processes that 
underlie situation awareness; and (3) the use of computer-based human performance 
models to augment the measures of situation awareness derived from performance data is 
explored. Finally, two potential example applications of the proposed assessment 
techniques are described, one concerning spatial awareness using wide field of view 
displays and the other emphasizing fault management in aircraft systems. 
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introduction 

The term situation awareness is in- 
creasingly finding its way into the litera- 
ture on the human factors of system de- 
velopment. In everyday parlance, it 
means the up-to-the-minute cognizance 
required to operate or maintain a system. 
It is generally agreed that expert pilots 
maintain intimate and meticulously up- 
dated knowledge of the state of their air- 
craft and its support systems. Although 
such situation awareness contributes to 
good performance, it is not synonymous 
with it. It is possible to have good situa- 
tion awareness and still not be a good pi- 
lot because of poor motor skills, 
coordination or attitude problems. 
Conversely, under automated flight 
conditions it is possible to have good 
performance with minimal situation 
awareness. Indeed, much of this 
knowledge may be superfluous in the 
case of routine flights. However, when 
emergencies arise and aids can no longer 
be relied upon, the completeness and 
currency of the pilot's situation 
awareness are critical to the ability to 
make decisions, revise plans, and 
manage the aircraft. 

In aviation communities there is in- 
creasing concern about whether the ap- 
propriate level of situation awareness is 
being acquired and maintained by crews. 
One focus of attention is on the extent to 
which automation that is designed to 
reduce crew workload serves inad- 
vertently to decrease their situation 
awareness. It is argued that as automa- 
tion takes over functions previously as- 
signed to the crewmembers, they lose 
contact with the information needed to 
perform and monitor these functions, the 
very information that is essential to ef- 
fective decision-making in the event of a 
malfunction or emergency situation. 

On the other hand, the advent of high 
technology or "glass cockpits" has 
opened up the design space to include 
"smart" information retrieval, large 
screen pictorial displays and even three- 
dimensional displays. These advances 
offer the potential to make it easier for 


the crew to acquire and maintain certain 
aspects of situation awareness. 

Why Measure Situation 
Awareness? 

Reason (1990), in his discussion of 
human error, distinguishes between 
"unsafe acts" and "precursors of unsafe 
acts." Our goal in developing techniques 
for measuring situation awareness is to 
help uncover information-processing 
precursors of unsafe acts. Analyses of 
scenarios that tax situational awareness 
may reveal shortcomings of user-aids 
that would not be evident in a less de- 
manding scenario. Similarly, the use of 
probes and experimental tasks may un- 
cover misconceptions and maladaptive 
processing strategies that are inconse- 
quential under normal circumstances, but 
could be detrimental in an emergency. 

The ultimate goals of designing and 
training to enhance situation awareness 
depend on having satisfactory quantita- 
tive methods for measuring it. The de- 
velopment of measuring instruments, 
however, requires that the cognitive pro- 
cesses involved in achieving and main- 
taining situation awareness be under- 
stood. Several seminal papers have ex- 
plored the information-processing de- 
mands on the commercial or military 
pilot (Beringer & Hancock, 1989; 
Endsley, 1988; 1989; Fracker, 1988; 
Regal, Rogers, & Boucek, 1988; Sarter 
& Woods, 1991; Whitaker & Klein, 
1988). 

The objective of this report is to ex- 
pand upon this work by: (a) developing 
a theoretical framework based on psy- 
chological theory for understanding sit- 
uation awareness, (b) using the frame- 
work to derive new strategies for assess- 
ing situation awareness experimentally, 
and (c) applying the proposed strategies 
to some sample assessment problems. 

Situation Awareness and Pilot 
Error 

The problem of maintaining situation 
awareness in the modern, semi-auto- 
mated cockpit has been brought to public 
attention through several dramatic inci- 


dents involving pilot error. One such in- 
cident involved a China Airlines 747 jet 
near San Francisco: 

The Taiwanese pilots, including a relief 
captain and flight engineer, had been flying 
nearly 10 hours, most of the time gazing at 
their instruments and watching the autopilot 
system navigate and fly the airplane. When 
the 747 went through a bit of turbulence, the 
auto throttle system, programmed to main- 
tain a constant airspeed, first pulled the 
throttles back to reduce power from the en- 
gines and then, about 30 seconds later, in- 
creased the power to prior levels, just as it 
was supposed to. However, the outboard 
engine on the right wing did not respond 
fully, and despite the flight engineer’s at- 
tempts to cope with the situation, the engine 
quit The 747 has four engines, so the loss 
of one engine at high altitude is not even 
considered an emergency and is easily han- 
dled. Had the captain completely turned off 
the autopilot and followed correct proce- 
dures, the flight would have continued with- 
out incident... 

The loss of the engine had caused the air- 
plane to try to turn to the right. Sensing this, 
the autopilot began deflecting the control 
wheel in the cockpit to the left attempting to 
stop the turn. With his attention focused, in- 
appropriately and almost exclusively on the 
engine problem, the captain failed, according 
to the N.T.S.B. report to realize that the air- 
plane and the autopilot had become engaged 
in a vigorous tug-of-war. The airplane was 
trying harder and harder to turn, or roll, to 
the right because of the dead engine, and the 
autopilot was struggling to prevent this and 
keep the airplane level, as programmed. 

Although the struggle between the two 
machines went on for nearly four minutes, 
the N.T.S.B’s analysis afterward found, the 
captain was entirely oblivious to it because 
he was letting the autopilot fly and did not 
actually have his hands on the control 
wheel.... Finally, he disconnected the au- 
topilot and took hold of the control wheel to 
fly the airplane himself. In that instant, the 
airplane immediately won the tug of war 
with the autopilot. The effect was similar to 
what happens in a real tug of war when one 
side, at a previously arranged signal, sud- 
denly lets go of the rope... The 747 rolled 
dramatically to the right.... The roll contin- 
ued and within a few seconds the 747 was on 
its back, plummeting earthward (Stockton, 
1988, p. 63). 


Fortunately, the incident ended with- 
out loss of life. The pilot recovered the 
plane at 10,000 feet, but not before the 
passengers had spent some time hanging 
upside down from their seatbelts. This 
incident provides an example of the 
problems that can occur when the trans- 
fer of control from automated equipment 
to human crewmember is carried out 
without appropriate awareness of the full 
situation. 

How Has Situation Awareness 
Been Defined? 

The China Airlines incident 
described above helps to make the point 
that many factors are involved in 
achieving situation awareness. It is not 
surprising, therefore, that researchers 
have yet to agree on a definition of it. 
Consider, for example, the following two 
definitions: 

[Situation awareness] means that the pilot 
has an integrated understanding of factors 
that will contribute to the safe flying of the 
aircraft under normal or non-normal 
conditions. The broader this knowledge is, 
the greater the degree of situational 
awareness (Regal, Rogers, & Boucek, 1988, 
p. 65). 

[Situation awareness is] the perception of 
the elements in the environment within a 
volume of time and space, the 
comprehension of their meaning and the 
projection of their status in the near future 
(Endsley, 1988, p. 97). 

These definitions, which are repre- 
sentative of those in the literature, differ 
in their emphasis on knowledge (i.e., 
product) and perception (i.e., process) re- 
spectively. Yet they both seem to 
capture the key features of situation 
awareness: an understanding of the 

meaning of events and the ability to 
anticipate the consequences of taking or 
failing to take particular actions. 

A Cognitive View of Situation 
AWARENESS 

We distinguish the process of achiev- 
ing situation awareness, which is 
sometimes called situation assessment, 
from its product or consequences. The 
state of awareness with respect to 
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information and knowledge is the 
product. The process, in contrast, 
involves an active and dynamic series of 
cognitive activities. Maintenance of 
situation awareness is not easy because 
the process requires mental resources 
that may be in competition with on- 
going task performance. The infor- 
mation gathering activities that 
contribute to situation awareness there- 
fore may heighten workload momentar- 
ily. However, a principal benefit of 
achieving situation awareness is that the 
operator or crewmember is prepared to 
deal with upcoming events such that the 
extreme surges in workload that can oc- 
cur in unexpected circumstances are 
avoided. 

The development of suitable mea- 
surement techniques requires an under- 
standing of the process by which situa- 
tion awareness is achieved within the 
workload constraints imposed by the 
multiple tasks confronting the pilot. In 
broad perspective, although the 
crewmembers spend much of their time 
in routine, repetitive activities, a number 
of different, potentially knowledge-in- 
tensive and procedurally-complex tasks 
may demand attention at any moment. 
Each of these tasks is usually triggered 
by a stimulus event, such as a communi- 
cation from Air Traffic Control (ATC) or 
a warning light, and, in order to obtain 
proper interpretation, may require that 
the crew initiate additional information- 
seeking behavior. The aircrew needs to 
be prepared for these tasks by having ad- 
equate understanding of the situation and 
knowing exactly what additional infor- 
mation is needed. For each such alerting 
signal or event, the crew must determine 
its relevance, its procedural and goal-re- 
lated implications, and its urgency. 
Especially for the experienced crew, 
such events may often call forth highly 
practiced patterns and result in 
"automatic" responses that do not add to 
the workload. If not, however, ass.-^ving 
their significance may require access to 
the full range of human memory struc- 
tures. The effort required to retrieve as- 
sociated data and knowledge necessary 


for deciding on a course of action neces- 
sarily results in an increase in mental 
workload. The framework we propose 
for understanding situation awareness 
draws on research and theory in cogni- 
tive psychology, particularly work on 
perception and cognition and on memory 
and attention. 

Neisser's Theory of the 
Perceptual Cycle 

Neisser (1976) has formulated a view 
of the integration of perception and cog- 
nition that provides a useful way of 
thinking about situation awareness for 
several reasons. First, it resolves the ten- 
sion between knowledge and perception 
that is evident in the above definitions of 
situation awareness. Building on Bartlett 
(1932) and Gibson (1979), Neisser ar- 
gued that knowledge, in the form of 
schemas, or mental models, leads to an- 
ticipations of certain kinds of informa- 
tion and directs attention and exploratory 
movements to particular aspects of the 
available information (See Figure 1). 
The information that the perceiver picks 
up from the environment, or samples 
from the available information, in turn, 
modifies, or updates, what the perceiver 
knows about the immediate surroundings 
and influences what is known about the 
world in general. Situation awareness, in 
this framework, can be seen as both 
product and process. As product, situa- 
tion awareness is the state of the schema. 
As process, situation awareness is the 
state of the perceptual cycle at any given 
moment. Situation awareness can be 
thought of as "the big picture" or context 
in which to interpret the flow of events. 
It allows the perceiver to attend to the 
right information at the right level of ab- 
straction for the right task. It facilitates 
the handling and scheduling of multiple 
tasks and prevents overload. Strategies 
for reducing workload in multiple task 
situations will be touched upon in a later 
section. 

Neisser's view of perception can also 
be expressed in terms of the new connec- 
tions theories arising out of work on 
parallel distributed processing systems 
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(Norman, 1986). In these theories, 
newly perceived features are thought to 
be associated with other, pre-existing 
features through excitatory or inhibitory 
connections based on past experience 
with those features. Maintaining situa- 
tion awareness, in these terms, is the op- 
timal adjustment of weights, or tuning of 
the system, to anticipate the arriving data 
with minimal discrepancy. 



Figure 1. Neisser's (1976) view of the 
perceptual cycle. Copyright © by 
W.H. Freeman and Company. 
Reprinted with permission. 

A second advantage to Neisser's ap- 
proach to perception is that it emphasizes 
the role of meaning or comprehension 
that was mentioned in both of the above 
definitions. Following Gibson (1979), 
Neisser handles the problem of meaning 
by making it an integral part of the per- 
ceptual cycle. 

The information picked up in vision is 
necessarily optical, consisting of patterns in 
the light over space and time. But optical 
information can specify objects and events at 
various levels of abstraction and meaning. 
When we perceive [a person's] mood, we are 
not engaged in the same perceptual cycle as 
when we are attending to his lip movements. 
We develop a different (though perhaps 
overlapping) set of anticipations; we pick up 


information that extends over a different 
span of time (Neisser, 1976, p. 21). 

A third advantage to this approach is that 
it handles the anticipatory processes that 
are mentioned in Endsley's (1988) 
definition and referred to as the ability to 
"think ahead of the aircraft" in Regal et 
al.’s (1988) discussion. In Neisser's per- 
ceptual cycle, the anticipation of events 
directs exploratory behavior. 
Exploratory behavior can occur in many 
different forms. Attention can be chan- 
neled to new objects with or without eye 
movements and with or without locomo- 
tion. Anticipation can occur in each of 
these situations. Anticipation can also 
occur at different levels in the continuum 
from perception to cognition. For exam- 
ple, at the basic perceptual level, antici- 
pation means a readiness to perceive cer- 
tain information, as when the optical 
flow specifies an impending collision or 
the occlusion of another aircraft by cloud 
cover. At the level of higher-order per- 
ceptual strategies, anticipation means fo- 
cusing attention efficiently, as when an 
experienced driver (or taxiing pilot) fo- 
cuses attention on a point at a distance 
from the vehicle rather than directly in 
front of it. At the cognitive level, antici- 
pation means the consideration of possi- 
ble outcomes. It includes knowledge-in- 
tensive activities such as contingency 
planning and diagnosis of system faults 
that may impact the flight. This level is 
best represented by Neisser's (1976) ex- 
panded view of the perceptual cycle (See 
Figure 2). The inner circle is the percep- 
tual cycle shown in Figure 1. The outer 
circle is a more general exploratory cycle 
that includes actions taken to obtain in- 
formation that is not present in the im- 
mediate environment. This kind of ex- 
ploratory behavior is based upon knowl- 
edge of the world and its possibilities. 
At each of these levels, anticipation 
serves to help in the management of 
workload. Attentiveness to potentially 
important parameters during periods of 
low workload makes the perceptual pro- 
cess more efficient during periods of 
high workload, thereby protecting 
against overload. 
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Figure 2. Neisser's (1976) exnanded view of the oerceotual cvcle. CoDvrieht © by 
W.H. Freeman and Company. Reprinted with permission. 


Real-World Constraints on 
Situation awareness 

Having borrowed Neisser's notion of 
the perceptual cycle to clarify what we 
mean by situation awareness, we now 

turn to the important question of why it 
sometimes is and sometimes is not 
achieved. To answer this question, we 
need to consider the task that the pilot is 
trying to accomplish, the nature of the 
information that is available, the inherent 
constraints on human processing capa- 
bilities, and the nature of expertise. 

The Pilot's Job 

The most striking characteristic of 
the pilot's job is the multiplicity of tasks 
that must be accomplished. For exam- 
ple, at 2500 feet during an approach to- 
wards a rain storm, the pilot may have to 
perform the following: 

• Monitor the descent (e.g., compare 
actual to desired flight path, airspeed 
etc.). 


• Perform the prelanding checklist. 

• Set the flaps/slats. 

• Receive radio messages from ATC to 
reduce speed and watch for traffic. 

• Enter the new altitude restriction into 
the Mode Control Panel (or equiva- 
lent). 

• Look out the window for traffic. 

• Respond to ATC about traffic seen or 
not ("seen" means pilot assumes re- 
sponsibility for separation). 

• Worry about the presence of 
windshear by looking for virga, 
lightning, or dust-rings. 

• Watch airspeed, especially for evi- 
dence of a windshear encounter. 

• Ignore radio traffic to other aircraft 
except where it contains warnings of 
nearby hazards. 

• Monitor the copilot's performance. 

Most of these tasks are triggered by 

specific events and many have a time 

frame during which they must be com- 
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pleted. Most require careful monitoring 
of information sources. 

The Available Information 

In carrying out these tasks, the pilot 
must be attentive to numerous sources of 
information. There is an ongoing stream 
of sensory information from the envi- 
ronment as well as cockpit visual and 
auditory displays, manuals, checklists, 
and communication from within or from 
outside the aircraft. Some of these in- 
formation sources can be consulted at 
will, others intrude at unexpected times. 
For example, the following sensory 
events, all of potential significance, may 
occur in sequence while the pilot is in 
the midst of an approach: 

* A flash of lightning. 

* An altitude glitch that results from 
turbulence. 

* Other signs of turbulence. 

* A radio transmission. 

* A speed reading that is right for 
flaps-down. 

For each of these data or events, the 
pilot must determine relevance, procedu- 
ral implications, and urgency. This de- 
termination can be more or less effortful, 
depending on factors that relate to hu- 
man information-processing constraints. 
These constraints will be discussed in the 
next section. 

Thus far we have described the mul- 
tiplicity of tasks facing the pilot and the 
multiple sources of information to which 
the pilot must attend. Now we will con- 
sider the factors that affect the degree of 
effort that the pilot must exert to deter- 
mine the significance of the incoming 
information. We will discuss two major 
factors: (a) the depth of the analysis re- 
quired and, (b) the extent to which atten- 
tion to the input requires a change in fo- 
cus or context 

Depth of Analysis Required 

In some circumstances, the pilot can 
choose to ignore or attend to incoming 
information on the basis of simple di- 
mensions, such as location or modality. 
Location within the cockpit provides in- 


formation about the function of certain 
inputs (e.g., primary flight displays are 
for control and guidance information, 
map is for navigation information). 
Cockpit location also provides informa- 
tion about the criticality of certain inputs 
(e.g., warnings are listed above cautions 
which are listed above advisories). 
Criticality is also specified by modality 
(e.g., the distinctive sound that signals a 
warning). 

While the relevance of some inputs 
can be determined easily, the relevance 
of others is not signalled by either source 
or modality, but depends on its relation- 
ship to some particular aspect of the 
flight situation. For example, in an ap- 
proach to landing, the significance of a 
radio message cannot be determined 
without further processing: a message 
from ATC to reduce speed and watch for 
traffic would certainly be relevant, a re- 
port from the preceding pilot would be 
relevant if it contained information about 
a potential hazard, a communication ad- 
dressed to another pilot would be irrele- 
vant, unless the pilot were nearby and it 
described conditions at the relevant air- 
port, and finally, a radio weather report 
would be relevant if it pertained to the 
right part of the country. 

While the significance of a warning 
or radio communication is relatively un- 
ambiguous, many of the inputs that pilots 
receive require additional information 
before their relevance can be determined. 
For example, a turbulence report en route 
, to an airport could signal weather diffi- 
culties at the destination or merely a tem- 
porary disturbance. A simulation study 
involving precisely these circumstances 
showed that only the best pilots took the 
implications of the turbulence seriously 
enough to seek information about cross- 
winds at the designated airport; as a con- 
sequence, only the best pilots were pre- 
pared for the task of rerouting (Orasanu, 
1990 ). 

To summarize, one factor that influ- 
ences whether and how easily an input 
can be processed has to do with the 
structure of the information. The signifi- 
cance of certain inputs is directly speci- 
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fled by superficial, attention-getting fea- 
tures while that of others requires deeper 
processing and further acquisition of in- 
formation. 

Change in Focus or Context 

The depth of processing required to 
determine the significance of an input is 
an important factor in determining 
whether or not it will be properly at- 
tended. Recent work on memory and at- 
tention suggests that another factor af- 
fecting the processing of an input is the 
extent of the competing demands on the 
pilot's attention at any given moment. 
The bottom line is that human informa- 
tion-processing capabilities are not well 
suited to a multiplicity of simultaneous 
tasks. Thoughtful attention is modular. 
People can consciously think about only 
one thing at a time. As a result, they do 
not handle interruptions and distractions 
very well. Research has shown that even 
when an operator is faced with as few as 
two tasks and the "tasks" consist of noth- 
ing more than the detection or recogni- 
tion of simple signals, the requirement to 
divide or switch attention between them 
may result in a significant loss in sensi- 
tivity or time that can be allocated to ei- 
ther (Broadbent, 1957; Schneider & 
Detweiler, 1988; Swets, 1984). 

In addition, research indicates that 
mental shifts between topics or semantic 
domains require measurable time and ef- 
fort and are prone to certain classes of 
biases and errors (Anderson & Pitchard, 
1978; Bower, 1982; Sanford & Garrod, 
1981; Schank et al., 1982). To the extent 
that incoming information is unrelated to 
the task in which the pilot is concurrently 
engaged, its interpretation must involve 
considerable mental workload. The 
more time and effort the pilot invests in 
its interpretation, the greater its potential 
for blocking notice or proper interpreta- 
tion of other available data. The less 
time and effort invested in its interpreta- 


tion, the greater the likelihood of mis- 
construing its implications. These ideas 
are captured in a theory about memory 
and attention that will now be described. 

RECONCEIVING THE DEMANDS ON 
THE PILOT 

A theory about memory and attention 
developed by Sanford & Garrod (1981) 
helps to clarify the cognitive demands on 
the pilot even though, interestingly, it 
was formulated to explain text compre- 
hension. We will present the theory as it 
was originally formulated, in terms of 
the comprehension of events in a story, 
and then show how it applies to the 
comprehension of events in a flight. 

Sanford and Garrod's Theory of 
Memory and Attention 

Sanford and Garrod (1981) theorized 
that an individual’s active memory con- 
sists of two bins, explicit and implicit 
focus. We find it convenient to think of 
these two bins as replacing the box la- 
beled "schema of present environment" 
in Neisser's (1976) expanded view of the 
perceptual cycle (See Figure 3). Explicit 
focus corresponds roughly to what is 
conventionally labeled as "working 
memory." At any given moment, ex- 
plicit focus contains a tightly- limited 
number of interrelated tokens of (or 
pointers to) larger knowledge structures 
in long-term memory. Although the 
contents of explicit focus are regulated 
more or less like a push-down stack, the 
maintenance of any given token depends 
not only on the recency with which it has 
been activated by the text or situation but 
also on its implicit relevance to the cur- 
rent interpretive stream. 

Implicit focus, in contrast, subtends 
the full-blown representation of the sit- 
uation that is partially represented in 
explicit focus. Information relevant to 
the knowledge in implicit focus cannot 
be brought to the interpreter's attention 
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Figure 3 . Modification of Neisser's (1976) view of the perceptual cycle 


with either the speed or the obliqueness 
of reference that suffices for information 
in explicit focus. On the other hand, 
such information can be interpreted far 
more quickly and with smaller increases 
in workload than information that is un- 
related to the contents of explicit focus. 

To support these active memories, 
Sanford and Garrod suggest that the 
reader's long-term (currently inactive) 
memory is also sectioned into two bins. 
We think of these bins as replacing the 
box labelled "cognitive map of the world 
and its possibilities" in Neisser's ex- 
panded view of the perceptual cycle (See 
Figure 3). The first, episodic memory, 
contains a complete record of the knowl- 
edge structures that have been built or 
accessed in the course of reading the cur- 
rent text or, for our purposes, in the 
context of the current flight. 

Meanwhile, semantic memory con- 
tains a person's lifetime accumulation of 
knowledge in general. Knowledge in ei- 
ther of these long-term memories can be 
brought to consciousness only given 


considerable effort or strong cueing. 
Requirements to do so are thus expected 
to be relatively costly in terms of mental 
workload. 

Extrapolating from the literature on 
text comprehension, we can anticipate 
some of the parameters that will control 
the ease or probability with which a 
given event is properly processed by the 
manager of real-world information. 
Because these ideas have not been tested 
or validated in the general domain of 
real-time information management, this 
extrapolation is speculative. 

Events that are relevant to those as- 
pects of a task on which a person is cur- 
rently working should be readily assimi- 
lated because they will map themselves 
onto the knowledge currently in explicit 
focus; thus, for example, the pilot will 
readily notice and respond to changes in 
the glideslope indication that occur in the 
course of landing (See Figure 4). Events 
that pertain to the task but not to the par- 
ticular aspect of the task with which a 
person is engaged are also expected to be 
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interpreted relatively quickly and co- 
gently as they will map onto knowledge 
in implicit focus; thus, for example, even 
while tracking the glideslope, the pilot 
may be readily alerted to changes in en- 
gine noise that are consistent or inconsis- 
tent with landing experience (See Figure 
4). In contrast, when the interpretation 
of an event requires consideration of 
knowledge in long-term memory, the 
probability or effort associated with its 
proper processing will depend on such 
factors as the transparency of its signifi- 
cance and the time available for working 
on it; thus, for example, very close to 
touchdown, the pilot will be relatively 
unprepared to receive and interpret unre- 
lated communications. 

Mission Categories 

We find it useful to think of the tasks 
that command the pilot's attention as 
falling into the following mission cate- 
gories: 

• Macro-planning and -navigation. 

• Local navigation, guidance and con- 
trol. 

• Communication outside the cockpit 

• Flight crew resource management. 

• Cabin management. 

• Routine management of physical 
equipment, resources and systems. 

• Routine management of Flight 
Management Computer (FMC) and 
related crew-aiding systems. 

• Troubleshooting of physical equip- 
ment, resources and systems. 

• Troubleshooting of FMC and related 
crew-aiding systems. 

• Bridging activities (breaks, teaching 
etc.). 

Each of these categories has specific 
goals associated with it, although the 
goals in any one category may serve as 
subgoals for another category. The work 
by Sanford & Garrod (1981) on text 


comprehension suggests further analo- 
gous constraints on pilot behavior. First, 
only one goal will be in explicit focus at 
a time. Data relevant to a new concern 
will more easily capture an operator's 
attention upon completion of a goal or 
subgoal. Although part of the goal 
structure may be available in implicit or 
explicit focus, the complete goal 
structure may be available only through 
long-term memory and thus require 
effort to retrieve. As a result, the 
necessity of switching to a task in 
another mission category before com- 
pletion of the first task could lead to less 
than optimal deployment of attention on 
either task. Switching to another task 
within the same category will be less dis- 
ruptive because the two tasks are likely 
to have a common goal and common im- 
plicit information. 

The effort required to switch 
attention between tasks in different 
mission categories means that the 
original task may be forgotten altogether, 
since the goal is no longer in working 
memory, especially if further distractions 
prevent the pilot from focusing on the 
goal structure long enough to reactivate 
the original task from long-term 
memory. Such memory lapses can be 
dangerous. 

An Airline Accident 

Many of the airline accidents that 
have been reported lend credence to the 
above depiction of the limits on human 
attention. An example is provided by a 
recent accident at the Detroit 
Metropolitan Airport (NTSB, 1988). 
The aircraft took off without setting its 
flaps and crashed. Although the crew 
had begun the pre-flight checklist 
properly (mission category: management 
of physical equipment, resources, and 
systems), they were interrupted by Air 
Traffic Control before verifying the 
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Figure 4. Hypothetical view of the pilot during landing. The boxes show the 
contents of memory; the circles show the objects of awareness. 


status of the flaps (mission catego^: 
communication outside the cockpit). 
Although they might still have resumed 
the checklist routine prior to take-off 
(management of physical equipment), 
other issues usurped their attention: they 
were confused as to which taxi-way to 
use (local navigation), the runway direc- 
tion had just been changed, and weather 
and runway conditions were not 
provided until the aircraft was already 
taxiing (macro-planning). 

Although, with proper handling, the 
aircraft could have become airborne 
without flaps, the crew had been given a 
windshear alert. When the problem with 
the flaps expressed itself during take-off, 
the symptoms were interpreted — and re- 
sponded to — as though they were caused 
by windshear (See Figure 5). As Figure 
5 illustrates, this misinterpretation is un- 
derstandable in light of the salience of 
windshear in the pilot's memory and the 
similarity of symptoms in the two cases. 
Very likely, the cost of this misinterpre- 


tation was compounded by what 
Norman, (1986) calls a tendency toward 
"cognitive hysteresis," the fact that hu- 
man information-processors are notori- 
ously bad at seeking disconfirming evi- 
dence once a hypothesis has been par- 
tially confirmed. Evidently the distrac- 
tions from the competing tasks left few 
resources with which to counteract this 
natural tendency to stand by an initial 
decision. The Detroit incident, in short, 
is typical of accidents resulting from 
human error. A series of minor errors, 
none of which was fatal in itself, to- 
gether resulted in a disaster (Reason, 
1990). 

The Nature of expertise 

Next we briefly consider how experi- 
enced pilots are able to perform the mul- 
tiplicity of tasks without error, in spite of 
inherent limitations on the human infor- 
mation processing system. The limita- 
tions are not so debilitating as they might 
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Figure 5. HvDOthetical view of the pilot during the incident described in the text. 

Expectations based on a windshear alert caused the pilot to overlook the 
true cause of the lift problem. 


first appear because of the extreme 
adaptability and resourcefulness of the 
human cognitive system. Schneider & 
Detweiler (1988) have identified seven 
types of compensatory strategies that 
people use to overcome performance 
limitations in multi-task situations. 
These strategies are thought to emerge 
routinely in the course of practice on 
multiple tasks, although the first two 
may be specific to conditions of high 
workload (Schneider & Detweiler, 
1988). 

Shedding, Delaying, and Preloading 
Tasks 

The first set of strategies consists of 
the following: 

Experience in performing multiple tasks 
enables one to anticipate and monitor the 
consequences of delaying or eliminaung a 
task (Schneider & Detweiler, 1988, p. 556). 

In the China Airlines accident, the 
pilot was preoccupied with looking out 
the window at the defective engine with 
the consequence that he delayed discon- 
necting the auto pilot, with disastrous 
results. 


Preloading involves preprocessing infor- 
mation prior to the onset of the critical 
workload segment (Schneider & Detweiler, 
1988, p. 556). 

One aspect of preloading, contin- 
gency planning, has been found to be 
characteristic of expert flight crews. 
Orasanu (1990) found that air crews who 
were rated as most successful in a simu- 
lated flight study (i.e., made the smallest 
number of errors) differed from crews 
rated less successful in the extent and 
timing of their contingency planning. 

Successful crews were likely to make 
plans (e.g., regarding weather closing of 
airports) and did so well in advance of 
when they were needed. The less suc- 
cessful crews, by contrast, were not pre- 
pared for the rerouting and ended up 
having to makes plans during a peak 
workload (e.g., while having to deal with 
a simultaneous hydraulics system fail- 
ure). 

The picture we get here is of good 
Captains who are planful, anticipate difficul- 
ties, use time during the normal phase to 
prepare for higher workload periods, and 
who are ahead of the curve. They stop and 
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think, then get information they need and 
carry out their plans. Emergency actions are 
rehearsed so the crew is primed to go into 
action when the time comes (Orasanu, 1990, 
p. 13). 

Letting Go of Unnecessary High- 
workload Strategies 

The expert finds short-cuts that 
reduce the workload. This aspect of 
pilot performance is particularly 
important when problems arise. The less 
effort required to monitor the flight, the 
more likely it is that the critical 
monitoring function will be maintained 
when the pilot has to deal simultaneously 
with a system failure. 

Utilizing Non-competing Resources 

This strategy involves learning to 
allocate task components to minimize re- 
source competition. The experienced pi- 
lot, for example, would be wary of carry- 
ing out another task that used spatial 
strategies while navigating, but would 
feel free to engage in activities that used 
quantitative strategies, such as 
calculating flying time. Successful 
cockpit resource management strategies 
also fall into this category. 

Multiplexing Transmissions Over 
Time 

This strategy has to do with optimiz- 
ing the division of attention across tasks: 

[Researchers] have shown that after 
extended training, human operators 
learn to sample instrument gauges at 
the optimal rate based on the relative 
information rate of each channel. The 
allocation of internal control process- 
ing may be tuned through experience 
in a manner comparable to the way 
the operators allocate attention among 
gauges (Schneider & Detweiler, 1988, 
p. 558). 

Shortening Transmissions 

This strategy consists of finding the 
minimal time that can be spent in 
looking at a gauge or source of 
information without sacrificing accuracy. 
Shorter viewing times also result, with 
experience, from learning to ignore 


irrelevant features and from chunking 
(See below). 

Converting Interference from 
Concurrent Transmissions 

This strategy involves finding ways 
to reduce interference from competing 
sources of information, by deliberate 
shifts of attention. Tuning out words 
spoken by your fellow crewmember 
when ATC is contacting you about 
something else is an example. The 
crewmember's message may be re- 
transmitted at a later time. 

Chunking Transmissions 

This strategy consists of grouping in- 
formation into related clusters to over- 
come the limits of working memory. 
Work on expertise in many domains has 
shown that the expert is more likely than 
the novice to perceive the higher order 
relations or patterns in the information 
that permit chunking. In a replication of 
an earlier work on chess masters, Chase 
& Simon (1973) found that experts, who 
were asked to reproduce the position of 
chess pieces from a real game after a 
brief glance, remembered more board 
locations than did novices. Their 
performance was not better than that of 
novices, however, when the pieces were 
randomly placed on the board. Thus, the 
experts were able to encode legitimate 
board arrangements into higher order 
units that could be perceived in a glance. 
In terms of flight management, we would 
expect more experienced pilots to differ 
from those with less experience in the 
ease and accuracy with which the 
patterns of information (i.e., "situations") 
derived from the displays could be 
perceived and remembered. 

Summary 

By way of review. Table 1 presents 
what we feel are the key elements of sit- 
uation awareness discussed thus far. The 
table is organized into three possible 
classes of situations faced by the pilot: 
routine, non-routine, and emergency. 
These classes all involve anticipating or 
"thinking ahead of the aircraft." This 
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ability becomes increasingly important 
as the situation changes from a routine 
flight to an emergency. Anticipation is 
accomplished in routine situations 
through the perceptual cycle depicted in 
Figure 1. Emergency situations, which 
require contingency planning and 
diagnosis, are accomplished through the 
larger exploration cycle, depicted in 
Figure 3. For each of the situations in 
Table 1, we specify the components that 
can require attention and the benefits of 
awareness. We then break down each of 
the situations into the relevant mission 
categories and summarize differences 
between novice and expert performance. 

Strategies for the Measurement 
of Situation Awareness 

In this section we address the prob- 
lem of assessing situation awareness. 
This problem has been discussed in sev- 
eral important papers (Beringer & 
Hancock, 1989; Endsley, 1988; 1989; 
Fracker, 1988; Regal, Rogers, & Boucek, 
1988; Sailer & Woods, 1991; Whitaker 
& Klein, 1988). The techniques that 
have been emphasized thus far are the 
probe technique and think-aloud proto- 
cols. In the probe technique, a simula- 
tion is expectedly or unexpectedly inter- 
rupted and the crewmember is asked to 
report on the state of various classes of 
information reflective of the state of his 
or her aircraft (Endsley, 1988; 1989). In 
think-aloud protocols, the crewmember 
is asked to provide a running narrative 
explanation of what she or he is doing 
during the execution of a scenario or 
during a post-flight debriefing. The need 
for reliable, sensitive measures suggests 
that we exploit these conventional ap- 
proaches, but that we also go beyond 
them in order to broaden the scope of 
measures and to capture more of the 
properties of both the process and prod- 
uct of situation awareness. 

In the remainder of this section, we 
discuss techniques for assessing situation 
awareness. The idea is to provide a 
range of strategies that can be fitted to 
different needs. The application that 
concerns us here is the evaluation of the 


situation awareness afforded by pro- 
posed new cockpit aids. However, we 
believe the strategies that we are 
proposing are general enough that they 
can be applied in areas other than avia- 
tion and to other assessment needs, such 
as the evaluation of the effects of train- 
ing programs or the development of cer- 
tification requirements. 

The strategies we will describe for 
measuring situation awareness fall 
into three categories: 

• Measures derived from scenario ma- 
nipulations : Situation awareness is 
inferred from performance on spe- 
cially designed scenarios that place a 
demand on crewmembers' attentional 
resources and are hypothesized to re- 
quire a sufficient state of awareness 
for good performance. 

• Direct measures: Situation aware- 
ness is inferred from a variety of 
probes and experimental tasks that 
tap the crew's knowledge and strate- 
gies during a scenario or following a 
brief vignette 

• Model-based measures : Situation 
awareness is inferred from the 
performance of a computer program 
that models the actions, knowledge 
states, and information processing 
strategies of a human (i.e., a flight 
crewmember) for a given situation. 

Although, we will discuss each of 
these measurement categories separately, 
they are intended to be used in combina- 
tion. 

Measures of Situation 
Awareness Derived from 
Scenario Manipulations 

We believe that with proper scenario 
design, much can be revealed about the 
situation awareness of the crewmember 
through performance measures alone. 
The key to designing scenarios for this 
purpose is to manipulate the demands on 
attention. Under normal circumstances, 
performance may not be a good indicator 
of how easily the pilot can stay abreast 
of the situation. A scenario in which 
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Table 1 

Charac teristics of Pilot Situation Awareness 

” — External 

Situation Benefit* of Mission Information 



Note: The rows are cumulative. In other words, the entries for emergency situations include all the entries 
for routine and non-routine situations. 
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events are entirely predictable and the ef- 
fort required to assess the situation is 
minimal is not likely to be be sensitive to 
differences in situation awareness. 
Therefore, scenario variations are needed 
that place competing demands on atten- 
tion. Attentional demands can be in- 
creased in several ways (See section on 
real-world constraints on situation 
awareness). Two that are relevant for 
the design of scenarios are: 

• Increase the depth of processing re- 
quired to comprehend the inputs. 

• Arrange for a shift in attention or a 

change of focus. 

These factors give rise, respectively, 
to the following strategies for assessing 
situation awareness. 

Design Scenarios with Subtle or 
Misleading Indicators 

The depth of processing required to 
interpret an input in a scenario can be 
manipulated by introducing either con- 
flicting, potentially misleading, or very 
subtle indicators. Situation awareness, 
or the lack thereof, can be inferred from 
the nature and timing of the choices 
made. The idea is that instruments and 
aids that provide good situation 
awareness will enable the crewmember 
to focus on subtle, but critical 
information and avoid being led down 
the "garden path" by information that is 
potentially relevant, but turns out to be 
inconsistent with the overall pattern. 
This approach was successfully applied 
to the evaluation of displays in a nuclear 
power plant control room by Woods, 
Wise, and Hanes (1982). 

An example of a scenario with subtle 
indicators is provided by Orasanu 
(1990). In one of the scenarios in her 
study, information about crosswinds at 
the airport provided definitive 
anticipatory information about a weather 
closing at the destination airport. This 
information was important because it 
allowed the crew to plan ahead while the 
workload was low. It was subtle, 
however, because it had to be solicited 
explicitly on the basis of another, less 


definitive indicator, turbulence en route. 
Only half the flight crews in her study 
solicited this information. In this 
example, the decision to seek 
information about crosswinds would be a 
good index of situation awareness. 

The Detroit accident, discussed ear- 
lier, is a clear case in which ambiguous 
information, concerning windshear, led 
to difficulties. Another is provided by 
the Midlands plane crash (NTSB, 1989). 
In this tragic incident, the pilot misinter- 
preted the indicators concerning an 
engine problem and turned off the good 
right engine instead of the faulty left 
engine. There were at least two 
indicators that helped to mislead the 
pilot, although there were other 
indicators that would have clarified the 
situation. First, there was smoke in the 
cockpit and smoke is usually associated 
with a problem in the right engine 
because of venting patterns. Secondly, 
there was a vibration that coincidentally 
ceased after the right engine was turned 
off, suggesting that the malfunction had 
caused the vibration. One could 
construct analogously ambiguous 
experimental scenarios and use the time 
and nature of subjects' decisions, e.g., to 
turn off an engine, as an index to their 
situation awareness. 

Design Scenarios with Interruptions 

Demands on attention can be 
manipulated in a scenario by interrupting 
the primary task for a period of time. 
The degree of disruption can be observed 
through the quality of the decisions that 
are made as a result of having to divide 
attention and the time needed to restore 
the original context after the disruption. 
The notion here is that cockpit aids that 
increase situation awareness should help 
the crewmember to deal with the 
competing demands on attention 
occasioned by an interruption. 

The Detroit accident again provides a 
case in point (See Figure 5). 
Interruptions from ATC while the crew 
was working on a checklist resulted in a 
failure to return to that task and, conse- 
quently, a failure to set the flaps — and 
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apparently to recall that the flaps had not 
been lowered — before take-off. This 
oversight had severe consequences be- 
cause other factors conspired to make the 
crew believe the problem was windshear 
and prevented them from taking the cor- 
rect actions. 

Although it is not difficult to intro- 
duce realistic interruptions into a sce- 
nario, for purposes of assessment it may 
be sufficient to interrupt the ongoing 
flow of information artificially by 
blanking the cockpit displays. With this 
paradigm, both the actions taken while 
"blind" and the effort required to restore 
the original context when the display 
returns can serve as an indicator of the 
extent to which cockpit systems have 
supported the development and up- 
dating of the crewmember's mental 
model or "cognitive map." Further, by 
altering certain data on "restoration," we 
can make finer assessments of the scope 
and focus of subjects' awareness. 

Direct Measures of Situation 
Awareness 

A number of more direct measures of 
situation awareness can be obtained by 
supplementing the scenario observations, 
discussed above, with experimental tasks 
and probes that shed light on the cogni- 
tive processes underlying the crewmem- 
ber’s performance. We will describe 
some assessment techniques that have 
proven to be robust within the field of 
experimental psychology and could be 
adapted to the problem of assessing sit- 
uation awareness. The four techniques 
that will be discussed are: 

• Queries. 

• Recall of inputs. 

• Category judgments (i.e., nor- 
mal/abnormal). 

• Information seeking/eye movements. 

Queries 

The technique of periodically query- 
ing the crewmember about situational in- 
formation, either during the scenario or 
at the end, is the most commonly used 
method for assessing situation awareness 
(Endsley, 1988; 1989). While this is a 


useful technique with high face validity, 
there has been little discussion about 
how to select probes for different 
situations. We would advocate a 
systematic approach to the use of probes 
that includes questions about information 
and indicators that vary systematically in 
their relevance to the task in immediate 
focus and to the overall status of the 
aircraft and flight plan. These would 
include spatial orientation, environment 
(weather, traffic), current goals and 
procedures, contingency plans, aircraft 
systems status, aircraft performance, 
diagnoses of malfunctions, and crew 
responsibilities and knowledge (See 
Table 1). 

Recall of Displays 

Another strategy for measuring situa- 
tion awareness is to test the ease with 
which the relevant details of a display 
can be committed to memory. The 
rationale for this technique comes from 
studies of expertise in a number of 
domains, most notably chess (Chase & 
Simon, 1973). The findings from these 
studies consistently show that experts 
have a better memory for the details of a 
display (e.g., the positions of playing 
pieces on a chess board), after a brief 
viewing, than do novices. The 
consensus from this research is that 
experts are able to perceive meaningful 
patterns in the display that reduce 
memory load. This conclusion has been 
supported by the additional finding that 
experts are no better than novices at 
remembering incoherent displays (e.g., 
randomly placed chess pieces). 

Because recall is strongly affected by 
meaningfulness, and the perception of 
meaningful events is the essence of 
situation awareness (Vicente, 1988), 
intentional recall of relevant displays can 
serve as an index of situation awareness. 

Category Judgments 

This technique involves measure- 
ment of the time taken to judge display 
inputs as "normal" or "abnormal." The 
rationale for this measure is similar to 
that for the recall measure. The same 
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skilled perception (i.e., situation aware- 
ness) that allows for "chunking" of the 
information, also should enable the per- 
ceiver to notice deviations from the ex- 
pected pattern. We envision this tech- 
nique being used with short vignettes 
that terminate with a frozen display. The 
viewer's task would be to rate the situa- 
tion as normal or anomalous as quickly 
as possible. Normal displays would 
contain readings that would be likely to 
occur. An abnormal display could be 
constructed for each normal display by 
changing one or more of the indicators to 
a value that could not occur in a real sit- 
uation. 

Information Seeking 

The last strategy consists of tech- 
niques for determining the information to 
which the crewmember is attending at 
any given time. For example, it can be 
used to determine whether and why a 
crewmember failed to look at the critical 
portion of a display in a scenario. One 
way of accomplishing this goal is to re- 
strict the available information sources to 
those that the person actively seeks. In 
other words, no information from dis- 
plays, communication sources, or flight 
aids would be presented unless it were 
specifically requested. The crewmember 
would be instructed how to obtain the 
needed information, perhaps through the 
use of a menu or touch screen. A possi- 
ble disadvantage of this technique is that 
the viewer might fail to solicit informa- 
tion that would have been attended had it 
been present. A more sophisticated, but 
labor-intensive way of accomplishing the 
same goal would be to record eye- 
movements during the scenario. 

Model-Based Measures of 
Situation Awareness 

We are proposing the use of model- 
based measures as a supplement to the 
scenario-based and direct measures dis- 
cussed above. The scenario-based mea- 
sures allow us to create circumstances in 
which situation awareness is particularly 
important and observe whether the 
crew's response is reflective of that 
awareness. The more direct measures 


allow us to observe those aspects of situ- 
ation awareness on which the crewmem- 
ber can report or that can be inferred 
from specifically designed performance 
measures. We propose the use of model- 
referenced performance measures in an 
effort to probe deeper into the state of 
the cognitive structures that support ef- 
fective situation awareness. 

The kind of human-machine perfor- 
mance model to which we are referring 
includes: (1) a scenario driver to set the 
context in which the behavior is to be 
measured, (2) a representation of those 
aspects of a system being controlled that 
impact on the human performance re- 
quirements, (3) a representation of the 
goals and tasks that the human machine- 
system team is required to perform, in- 
cluding descriptions of the procedures by 
which they are performed and (4) repre- 
sentations of primitive human perfor- 
mance capacities and limitations, such as 
visual scanning rules, the state of infor- 
mation in memory, methods for assign- 
ing task priorities, and resource-based 
workload limits. 

Examples of such models are the 
PROCRU model that describes the be- 
havior of the pilot-flying and pilot-non- 
flying of a Boeing 727 accomplishing a 
final approach and landing (Baron, 
Zacharias, Muralidharan, and Landcraft, 
1980), and the AIRT model that de- 
scribes the behavior of the controllers in 
an Air Force forward air control center. 
(Corker, Cramer and Henry, 1990). 
While human performance models of 
this kind have been and are being devel- 
oped, the approach we are suggesting is 
speculative, since they have not previ- 
ously been used in support of real-time 
performance analysis in tandem with an 
on-going simulation test of human per- 
formance. However, we believe the 
state-of-the-art has advanced to the point 
that exploring such an application is now 
a productive research strategy. 

To apply models for the assessment 
of situation awareness in flight deck 
simulation studies, one would build a 
model of the flight deck control and dis- 
play environment, of the behavior of the 
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aircrew that was valid for the flight crew 
tasks under study, and a simplified 
model of the aircraft performance. The 
model would be run in parallel with the 
actual aircrew participating in the simu- 
lation trial. It would be set up with the 
same scenario background and flight 
plans. Information received by the hu- 
man crew would also be input to the 
modelled crew. This information would 
include data about the state of the aircraft 
and its systems that is available from 
flight deck displays and the content of 
ATC communications. Modelled crew 
actions would affect the modelled air- 
craft state. 

The modelled system would be regu- 
larly up-dated with information from the 
actual on-going flight simulator trial to 
insure that the modelled system did not 
get out of synchronization with the real- 
time performance of the human crew in 
the simulated aircraft. To the extent that 
the modelled system is a valid represen- 
tation of the real system, then the crew 
behavior will mimic the behavior of the 
human crew. 

When a successful mimic is 
achieved, then the analyst has available 
in the model a representation of the in- 
ternal state of the crewmember at each 
moment in time. This representation is 
exactly what is required to undertake an 
in-depth analysis of the state of the 
crewmember's situation awareness. At 
critical instants one could examine the 
current contents of the crewmember's 
memory representation; one could in- 
spect and evaluate the priority ordering 
of the queue of tasks waiting to be ac- 
complished. Tracking the contents of 
memory and task execution protocols 
would provide clues to points where lack 
of situation awareness is due to exces- 
sive monitoring requirements or task 
loading. Similarly it will highlight the 
source of differences in awareness that 
can be attributed to different display con- 
figurations or approaches to automation. 
In short it can provide substantial aug- 
mentation to situation awareness data 
without disrupting the conduct of the live 


simulation itself or relying on the self re- 
ports of crew-member subjects. 

ASSESSING THE IMPACT OF NEW 

Technologies on Situation 
Awareness: Two Examples 

To conclude, we present two exam- 
ples of how these strategies for assessing 
situation awareness can be used to evalu- 
ate the effect of proposed cockpit aids. 
The first example concerns a panoramic 
display that is intended to increase spa- 
tial awareness during landings. The sec- 
ond concerns a fault finder designed to 
increase the aircrew's awareness of sys- 
tem malfunctions. 

Example I: Evaluation of 
Panoramic Flight Path displays 

An evaluation of a proposed new 
cockpit display that provides a 
panoramic, 70 degree field of view of the 
landing strip under instrument flight 
conditions is currently being undertaken 
by Russell Parish's group at NASA- 
LaRC. The display depicts the intended 
flight path as a row of goal posts along 
an imaginary pathway in the sky. A per- 
spective view of a series of rectangles 
shows the position of the plane with re- 
spect to the intended flight path. At a 
point when the landing strip would come 
into view out the window, it is repre- 
sented on the display in its proper per- 
spective. Other aircraft operating in the 
near-by airspace are also indicated and 
are enclosed in an attention-getting red- 
outline box if they become a collision 
threat. 

This panoramic display is being com- 
pared to the use of conventional instru- 
ments, including a plan-view navigation 
display and conventional attitude dis- 
play. Special scenarios consisting of a 
final approach and landing are being 
used for this purpose. The investigators 
are interested in how each of the displays 
affects the pilot's ability to maintain spa- 
tial awareness and control of the aircraft 
in a range of circumstances. We de- 
scribe their assessment design here be- 
cause it is illustrative of many of the 
strategies we have suggested. 
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Introduce Subtle or Misleading 
Information 

In some scenarios, a difficult flight 
path with several turns will be required. 
At the most difficult point in the flight 
path, another aircraft in the vicinity will 
unexpectedly turn onto a collision course 
with the aircraft under control of the 
• pilot under study. The measure of 
situation awareness will be the time to 
detect the offending aircraft and the 
nature of the actions taken to correct it. 

Introduce Interruptions 

During other scenarios, the screen 
will go blank for five seconds and the 
plane will return to an unexpected loca- 
tion somewhat off the flight path. The 
pilot will be required to assess the new 
location and execute suitable maneuvers 
to return to the intended flight path. The 
measure of situation awareness is the 
time needed to initiate corrective action 
and the nature of any maneuver errors 
committed in returning to the intended 
path. It seems likely that the panoramic 
display will greatly facilitate this task, 
but quantitative measures of the differ- 
ences in display conditions will be illu- 
minating. Another potential measure 
would be the degree to which subjects’ 
maneuvering of the plane during the time 
the screen was blanked conformed to the 
original flight path. Close conformity 
under the panoramic display condition 
would suggest that the display had fos- 
tered the development of a good mental 
model that allowed for accurate predic- 
tion of the intended flight path. 

Queries 

Direct measure of spatial awareness 
will be obtained by interrupting some of 
the scenarios at unexpected points and 
asking subjects questions about the loca- 
tion of their aircraft and other aircraft in 
the area. The evaluation of spatial 
awareness, as illustrated by this study, is 
perhaps the most straightforward appli- 
cation of our approach. Yet it has re- 
quired considerable creativity on the part 
of the evaluation team. The assessments 


of situation awareness outside of the 
spatial domain, we suspect, will require 
even more ingenuity. For purposes of 
discussion, in the next example we con- 
sider how we would go about evaluating 
situation awareness regarding system 
malfunctions. 

Example 2: Fault-Finder 

Much thought is going into the de- 
sign of fault-finding aids. The Midlands 
Airline accident (NTSB, 1989), in which 
a crewmember erroneously turned off the 
good, right engine instead of the defec- 
tive, left one causing the airplane to 
crash on landing when the left engine 
failed, has provided a cogent example of 
a case where such an aid might have 
been advantageous. Although the air- 
crew consulted the relevant instruments 
before making the decision, they appar- 
ently did not interpret them correctly. 
For purposes of this discussion, we con- 
sider how we would go about comparing 
some of the new aids that are being de- 
veloped to enhance awareness of system 
malfunctions (e.g., Abbott, 1990; 
Hudlicka, Corker, Schudy, & Baron, 
1989). These aids perform one or more 
of the following functions: indicate 

whether instrument readings for various 
system parameters are within the 
expected range, produce hypotheses 
about possible malfunctions, and suggest 
procedures for dealing with the 
malfunction. 

We would first design some scenar- 
ios in which a malfunction occurred and 
observe crew performance with the dif- 
ferent flight aids. To increase the sensi- 
tivity of our measures, we would intro- 
duce factors into the scenario that in- 
crease the difficulty of attaining situation 
awareness: 

Introduce Subtle or Misleading 
Information 

The Midlands accident (NTSB, 
1989) provides some useful suggestions. 
The accident report points to problems of 
misleading and subtle information. 
Misleading information (the coincidental 
cessation of vibrations and smoke when 
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the good engine was turned off) caused 
the crew to think they had turned off the 
defective engine and to disregard the in- 
strument readings. Subtle information 
(the 16 degree roll to the left after the au- 
topilot was turned off but before the right 
engine was throttled back) was not no- 
ticed by the pilot, although it should 
have provided an indication that the left 
engine was not functioning properly. 

Introduce Interruptions 

We would again turn to the Midlands 
incident for inspiration. The flight crew 
in that scenario was diverted by the fol- 
lowing string of interruptions: the need 
to communicate with the passengers via 
the cabin address system, a weather in- 
formation advisory, the need to repro- 
gram the flight management system, 
ATC messages concerning a new radar 
heading, further descent clearance and a 
new radio frequency, checklist require- 
ments, a radio call requesting that a test 
call be placed to the aerodrome fire ser- 
vice, and distracting radio calls from 
other aircraft on the same frequency. 
These interruptions may have played a 
significant role in the incident because 
they prevented the pilot from returning 
to a conversation with the copilot in 
which he was just beginning to question 
the evidence that had led them to shut 
down the wrong engine. 

Second, we would use several kinds 
of direct measures of situation aware- 
ness, as discussed above: 

Queries 

We would ask questions to probe the 
crewmember's understanding of various 
aspects of system functioning either dur- 
ing or after the scenario. The accident 
report from the Midlands incident, in its 
thoroughness, provides a model of how 
to accomplish this task. In the aftermath 
of that incident, information was ob- 
tained from the crewmembers about: the 
meaning of the indicators — how reliable 
the readings are on this and other aircraft 
(this question was important because vi- 
bration readings which were accurate in 
this aircraft had been unreliable on other 


aircraft); the meaning attributed to vari- 
ous symptoms, such as smoke in the 
cabin (this question was important be- 
cause this aircraft was different from 
many others, in which only the right en- 
gine ventilates into the cabin); or the 
failure of the right engine to restart when 
it was engaged at the last minute (the en- 
gine start levers, or fuel valves, were 
found in the cutoff position after the 
accident). 

Recall of Displays 

One problem with the instruments for 
indicating engine malfunctions that be- 
came clear in the Midlands accident is 
that it was difficult to perceive patterns. 
Recall of the information in briefly pre- 
sented displays, therefore, would be one 
measure that would be important for as- 
sessing the effectiveness of the different 
aids. 

Category Judgments 

Another way of assessing the mean- 
ingfulness of the various displays would 
be to present displays of information for 
both right and left engines, where either 
one, both, or neither was defective, and 
have pilots make judgments as to which 
were abnormal, as rapidly as possible. 

Information Seeking or Recording of 
Eye Movements 

Finally, we would supplement the 
above measures, if possible, with the 
recording of eye movements to ascertain 
how frequently and how early the aids 
were consulted and how rapidly informa- 
tion was extracted from them. 

AUTOMATION AND SITUATION 
AWARENESS 

As has been suggested, one of the 
motivations for measuring situation 
awareness today is the interest in making 
the introduction of new cockpit aids and 
automation concepts more human cen- 
tered. In this section we elaborate on the 
problems that automation can introduce 
and the potentially new situation aware- 
ness requirements it invokes. 
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In the past the pilot obtained situa- 
tion awareness by doing, by being di- 
rectly involved in every aspect of the 
flight. Information that supported situa- 
tion awareness was a combination of that 
which the pilot was compelled to seek in 
order to be able to perform a task ade- 
quately and the feedback that occurred as 
a result of the pilot's actions. Situation 
awareness was enhanced by the lack of 
complexity. The danger, however, was 
that the manual workload of controlling 
the aircraft could be so high under cer- 
tain conditions that the extra attention 
that was required to construct good sit- 
uation awareness was not available. 

Automation can, and has relieved the 
flight crew of many functions, thus low- 
ering selected aspects of workload. It 
could be argued that by using automation 
the pilot has more time to actively seek 
the information that allows him to be sit- 
uationally aware. As we have argued in 
previous sections, achieving situation 
awareness is an active process of seek- 
ing, prioritizing and interpreting infor- 
mation. Successful accomplishment in- 
curs a workload penalty; someone who 
has no spare workload capacity because 
he is manually controlling attitude, 
speed, etc., will not have the same ability 
to achieve situational awareness as 
someone who has off-loaded part of the 
manual workload to automation. 

However, this eliminates the appar- 
ently natural ability to construct situation 
awareness "by doing". As automated 
systems have been introduced, the pilot 
has become more and more removed 
from certain aspects of the flight. He no 
longer is privy to all the raw data re- 
quired to adequately perform a task 
when that task is performed wholly or in 
part by the automation (even when he is 
responsible in the case of a failure). By 
the same token, the feedback given to the 
automation, which it requires in order to 
assess its dynamic performance, and the 
inputs the automation must make to 
compensate for a changing situation, are 
not always passed along to the pilot be- 
cause they are considered irrelevant. 
However, they can be highly relevant 


when the pilot is asked to take over on 
short notice in the event of a failure. 

In addition, the introduction of au- 
tomation adds many more information 
elements and interactions that one must 
keep track of. That is, each new piece of 
automation becomes part of the situation 
of which one must be aware. When a 
pilot manually controls the attitude of the 
aircraft, he must be aware of the chain of 
effects of stick movement, control sur- 
face movement, and the dynamics of re- 
sulting attitude changes. When the au- 
topilot is used, the pilot should still be 
aware of these items and often is not. 
Further, in the case of an unexpected 
event, he should be aware of the charac- 
teristics of the automation itself, such as: 
current autopilot mode, if there is more 
than one; factors or actions that will 
cause it to disengage; the values that 
drive it; and what happens if it fails. 
This all puts additional situation aware- 
ness requirements on the flight crew, 
which technology-driven automation of- 
ten does not make salient and when it 
does, it is at times when the human pilot 
is least able to cope with workload 
demands of the unexpected event 

As a result, any kind of automation 
that requires direct crew interaction in ei- 
ther routine or back-up mode operation 
expands the situation of which the pilot 
must be aware. It adds more items to the 
queue of information that must be 
tracked and it adds layers of complexity 
or abstraction to the situation so that the 
workings and nature of the remote layers 
are less observable. 

Automation that is to be human cen- 
tered must use the information manipula- 
tion, communication and presentation ca- 
pabilities associated with it to provide in- 
formational support and feedback about 
the state of the automation itself in an 
expeditious manner. In the ideal case, 
this cannot only compensate for the loss 
of direct feedback and simplicity, but can 
provide the pilot with perspectives that 
were unavailable without the automation 
(See Vicente & Rasmussen, 1990 for an 
example in the domain of process con- 
trol). Human-centered automation offers 
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the potential of greater pilot situation 
awareness than either no automation or 
technology-driven automation, which, in 
turn, should lead to improved overall 
human- system performance and aircraft 
safety. 

Crew Awareness 

The focus of the discussion of this re- 
port has been on the situation awareness 
of an individual crewmember, the pilot, 
the co-pilot or where applicable the 
flight engineer. There is also beginning 
to be a literature on the topic of crew 
awareness by which we mean the aware- 
ness of the aggregate set of individuals 
responsible for the safety and effective- 
ness of the flight (Wellens, 1989; 1990). 
Flight procedures identify that some 
features of awareness are required of the 
pilot-flying while others are required of 
all crewmembers. 

Assessing crew awareness focuses 
attention on the communication pro- 
cesses among crewmembers. 
Information on flight instruments is 
available to both crewmembers; however 
at critical moments one is usually attend- 
ing to out-of-cockpit sources while the 
other is "flying the instruments". When 
one is completing a check list, it is im- 
portant that the other be aware of non- 
routine findings when they have a po- 
tential impact on current or future deci- 
sions. A lack of crew awareness may 
have played a role in the Midlands inci- 
dent as well. Evidently, the flight atten- 
dants who saw flames coming out of the 
left engine were unaware that the pilot 
thought the right engine was the defec- 
tive one. To make matters worse, the pi- 
lot was unaware that the flight attendants 
who had seen the flames in the left en- 
gine, had not heard his announcement. 
As was discussed earlier, experienced 
and effective crews discuss precautions, 
although they may never need them, 
while workload is low so that they have 
little need for discussion when a problem 
arises. Less effective crews, by contrast, 
wait until a critical need arises before 
they begin to communicate, making for a 
very high workload (Orasanu, 1990). 


As we begin to think about the intro- 
duction of a computer-based intelligent 
flight assistant in the cockpit, the concept 
of crew awareness is further complicated 
because there may be critical information 
that this system takes account of that is 
not made available to the human 
crewmembers. The design of cockpit 
displays for such an assistant must con- 
sider the needs for crew awareness as 
well as for particular individual 
crewmembers. 

The objective measurement of crew 
awareness is only beginning to be ad- 
dressed. It will require the simultaneous 
and independent assessment of the 
awareness of each crewmember, using 
one or more of the methods we have dis- 
cussed, and the correlation of the indi- 
vidual results. It may also require the 
scoring of the content of communication 
messages to identify the points in time at 
which information is shared. 

SUMMARY AND CONCLUSIONS 

Situation awareness, as a feature of 
aircrew performance, is receiving in- 
creased attention as new display tech- 
nologies become available and higher 
levels of automation are being proposed 
for flight deck implementation. As a 
brief, comprehensive definition we quote 
Endsley, 

"The perception of the elements in 
the environment within a volume of time 
and space, the comprehension of their 
meaning and the projection of their sta- 
tus in the near future" (Endsley, 1988). 

Measuring situation awareness re- 
quires understanding the processes un- 
derlying the achievement of it as well as 
the kinds of informational "products" 
that result. These processes are charac- 
terized by a range of cognitive activities 
from automatic effortless processing to 
complex effortful inferential reasoning. 
Successful achievement of situation 
awareness may also be studied by exam- 
ining the different methods used by ex- 
perts to minimize the impact of exces- 
sive attention demands. 
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We emphasize three approaches to 
measurement strategy: (1) creative de- 
sign of scenario contexts that emphasize 
the demand for the aircrew's attentional 
resources and the need to interpret subtle 
and sometimes misleading clues, (2) di- 
rect probes and experimental manipula- 
tions designed to tap the knowledge and 
strategies used by the aircrew during a 
scenario or brief vignette, and (3) infer- 
ences from a computer program that 
models the actions, knowledge states and 
information processing strategies of the 
crewmember. 

The design of flight deck-related au- 
tomation can have many subtle effects 
on the requirements for and ease of 
achieving situation awareness because, at 
once, it adds elements to the array of 
systems that must be monitored at the 
same time that it takes away information 
that previously was obtained by direct 
interaction with the aircraft. Therefore, 
improved design of human-centered au- 
tomation can be facilitated by systematic 
objective measurement of the situation 
awareness associated with new designs. 
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